# File-Validation: MD5

**Note:** As mentioned in "[Security / Reliability](#security--reliability)", MD5 *does* have some flaws.
If you are looking for an algorithm, considered more secure, you may want to check out [Secure Hash Algorithm 2](https://notabug.org/jayvii/Tutorials/src/master/Security/Integrity/File-Validation:%20SHA-2)
Also, the SHA-2 tutorial is a follow-up to this one.


## Table of Contents
1. [Introduction: MD-WHAT?!](#introduction-md-what)
	* [Checksums](#checksums)
	* [Message Digest 5 (MD5)](#message-digest-5-md5)
	* [Other types of Checksums](#other-types-of-checksums)
	* [Security / Reliability](#security--reliability)
2. [Checksums - Why would I care?](#checksums---why-would-i-care)
3. [MD5 on Linux](#md5-on-linux)
	* [MD5-check on a CD](#md5-check-on-a-cd)
4. [MD5 on other OS's](#md5-on-other-oss)
	* [MD5 on Windows](#md5-on-windows)
	* [MD5 on OSX](#md5-on-osx)
	* [MD5 on BSD](#md5-on-bsd)
	* [MD5 on Solaris](#md5-on-solaris)
5. [Sources / Referal Links](#sources-and-referal-links)



## Introduction: MD-WHAT?!

### Checksums
In order to (roughly) explain, what an MD5-checksum is and why we need it, we first have to understand, what "checksums" are in general.
A Checksum (or "hash sum") is a small footprint from a block of data. It is used to detect corruption or change of that data, which may have happened during its transmission, storage or tempering. Checksums are a great way of verifying data integrity, but are unreliable to verify data authenticity (act of confirming the truth of an attribute, claimed true by an entity), which is what private keys are for (PGP/GPG)

### Message Digest 5 (MD5)
The MD5 algorithm is a cryptographic hash-function, which performs operations on the data (message) to compute a hash, consisting of 128 bits.
It is constructed so even one bit of information-change of the data leads to an extreme difference of the output-hash. This makes it difficult (though, not impossible) to create the same output-hash with two different input-files.

### Other types of Checksums
There are a whole lot of other hash-functions, each with pro's and con's, depending on field of usage, degree of difficulty, and the actual operation. The Most common (and usually sufficient) ones are:
* MD5
* SHA-1, SHA-2, SHA-3
You can learn a whole lot more about the theory behind and characteristics of each of these, if you look up "Cryptography", "Data-Validation" and "Data-Integrity" in specialized literature.

### Security / Reliability
Several sources show, that MD5 has a bunch of design-flaws. As such, it is not recommended for use in high security environments or businesses. However, it may be considered sufficient for the usage shown in this Tutorial/Article. However, you should be aware of MD5's flaws and risks. It is often advised the use different algorithms like [SHA-2](https://notabug.org/jayvii/Tutorials/src/master/File-Validation:%20SHA-2).
Further information can be retrieved from following sources:
* [Xiaoyun Wang, Hongbo Yu - How to Break MD5 and Other Hash Functions](http://merlot.usc.edu/csac-f06/papers/Wang05a.pdf)
* [J. Black, M. Chochran, T. Highland - A Study of the MD5 Attacks](https://www.cs.colorado.edu/~jrblack/papers/md5e-full.pdf)
* [M. Stevens, A. Lenstra, B. de Weger - Vulnerability of software integrity and code signing applications](https://www.win.tue.nl/hashclash/SoftIntCodeSign/)
* [A. Sotirov, M. Stevens, J. Appelbaum, A. Lenstra, D. Molnar, D.A. Osvik, B. De Weger - MD5 considered harmful today](https://www.win.tue.nl/hashclash/rogue-ca/)
* [J. Stray - web browser flaw could put e-commerce at risk](http://www.cnet.com/news/web-browser-flaw-could-put-e-commerce-security-at-risk/)
* [CMU Software Engineering Institute - Vulnerability Note VU#836068](https://www.kb.cert.org/vuls/id/836068)


## Checksums - Why would I care?
As written above, checksums/hashes can be extremely useful to verify/validate data, especially if downloaded from the web.
Corrupted data can be both: a security and stability risk, so it is very well advisable to check for this, whenever possible.

In fact, every respectable software-maintainer also provides some sort of hash to validate your file after downloading. At least in the opensource world, this usually means SHA-2 (SHA256) or MD5.
Not only can you check your downloaded OS-images, but usually, also your packagemanager compares hashes of each downloaded binary (or sourcecode) with the ones, provided in the repository.

Sadly, when you move away from opensource projects, this becomes a lot less common, making the downloaded files more vulnerable to tempering or damage.

**TIDBIT:** Often, 3rd-party download-suits or Torrent-Clients check hashes themselves, if provides by the uploader.


## MD5 on Linux
For the sake of this Tutorial/Article, I'll use Debian-8.2 (current version, as of writing this) as an example.
The file "*[debian-live-8.2.0-amd64-standard.iso](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-8.2.0-amd64-standard.iso)*" is available from one of Debian's official mirrors, which ALSO provides several textfiles with chechsums ([MD5](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/MD5SUMS), [SHA-1](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/SHA1SUMS), [SHA-2, 256-bit](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/SHA256SUMS), [SHA-2, 512-bit](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/SHA512SUMS)), as well as authenticity-checks for each of these.

The easiest way of checking the MD5-Hash on linux, would be the commandline-tool *"md5sum"*, which should be available (probably pre-installed as part of the base-system) to every Distribution out there.
In order to receive the hash of a file, you simply need to issue following command "**md5sum /PATH/TO/FILE**"
In our example, this could be:
`md5sum ~/Downloads/debian-live-8.2.0-amd64-standard.iso`

This will result in a 32-diget output, consisting of numbers and lower case letters, similar to this:
`fcd8835aab844ee5a8c2adcc23937107 debian-live-8.2.0-amd64-standard.iso`

This hash can now be compared to the appropriate string in the [MD5SUMS-file](http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/MD5SUMS), provided on the download-mirror.


Of course, it can be rather cumbersome to compare the hash all by yourself, especially if you have to do it for multiple files. md5sum does however provide a way to compare the hash automatically.
For this to work, the file you want to check needs to be in your current working directory (For example "~/Downloads/"). Assuming BOTH (the MD5SUMS-textfile, as well as the actual ISO) are located in your Downloads-folder, you can verify the ISO's integrity by adding the *-c* parameter and using the MD5SUMS-textfile as input, instead:
`cd ~/Downloads`
`md5sum -c MD5SUMS`

Your output will probably be a wall of text with a lot of errors in it (assuming, you only downloaded the ISO mentioned above). This is because md5sum will check for EVERY file listed in MD5SUMS. They key-part however is, the line "*debian-live-8.2.0-amd64-standard.iso: OK*" indicating that our downloaded file is in fact not damaged.
```
md5sum: debian-live-8.2.0-amd64-cinnamon-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-cinnamon-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-cinnamon-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-cinnamon-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-cinnamon-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-cinnamon-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-cinnamon-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-cinnamon-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-cinnamon-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-cinnamon-desktop.iso.zsync: FAILED open or read
md5sum: debian-live-8.2.0-amd64-gnome-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-gnome-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-gnome-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-gnome-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-gnome-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-gnome-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-gnome-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-gnome-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-gnome-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-gnome-desktop.iso.zsync: FAILED open or read
md5sum: debian-live-8.2.0-amd64-kde-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-kde-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-kde-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-kde-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-kde-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-kde-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-kde-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-kde-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-kde-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-kde-desktop.iso.zsync: FAILED open or read
md5sum: debian-live-8.2.0-amd64-lxde-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-lxde-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-lxde-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-lxde-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-lxde-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-lxde-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-lxde-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-lxde-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-lxde-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-lxde-desktop.iso.zsync: FAILED open or read
md5sum: debian-live-8.2.0-amd64-mate-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-mate-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-mate-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-mate-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-mate-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-mate-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-mate-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-mate-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-mate-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-mate-desktop.iso.zsync: FAILED open or read
debian-live-8.2.0-amd64-standard.iso: OK
md5sum: debian-live-8.2.0-amd64-standard.iso.contents: No such file or directory
debian-live-8.2.0-amd64-standard.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-standard.iso.log: No such file or directory
debian-live-8.2.0-amd64-standard.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-standard.iso.packages: No such file or directory
debian-live-8.2.0-amd64-standard.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-standard.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-standard.iso.zsync: FAILED open or read
md5sum: debian-live-8.2.0-amd64-xfce-desktop.iso: No such file or directory
debian-live-8.2.0-amd64-xfce-desktop.iso: FAILED open or read
md5sum: debian-live-8.2.0-amd64-xfce-desktop.iso.contents: No such file or directory
debian-live-8.2.0-amd64-xfce-desktop.iso.contents: FAILED open or read
md5sum: debian-live-8.2.0-amd64-xfce-desktop.iso.log: No such file or directory
debian-live-8.2.0-amd64-xfce-desktop.iso.log: FAILED open or read
md5sum: debian-live-8.2.0-amd64-xfce-desktop.iso.packages: No such file or directory
debian-live-8.2.0-amd64-xfce-desktop.iso.packages: FAILED open or read
md5sum: debian-live-8.2.0-amd64-xfce-desktop.iso.zsync: No such file or directory
debian-live-8.2.0-amd64-xfce-desktop.iso.zsync: FAILED open or read
md5sum: WARNING: 34 listed files could not be read
```


In order to improve visibility, you can filter out critical errors by adding *"2>&1"* at the end of the command. Piping that output through *"grep"* will only show successful hash-checks. Again, I'm assuming both files are located in your Downloads-folder (and I will continue assuming so from now on).
`cd ~/Downloads`
`md5sum -c MD5SUMS 2>&1| grep OK`

The output will look **a lot** cleaner, this time:
`debian-live-8.2.0-amd64-standard.iso: OK`


### MD5-check on a CD
Not only can downloading break a file, but also transfering it from one location to another **locally**. In the example of downloading a Distribution's ISO, that also means, that burning it to a CD or transfering it to a thumb drive may break it. A first thought would be, to run an MD5-check on the CD afterwards, like so:
"*md5sum /dev/cdrom*"
However that will **NOT WORK**!

You can use "dd", to read the cd/thumbdrive and pipe its output into "md5sum", though.
For this to work, we need the exact size of the ISO. For this, we simply use `ls -l`.
We pass the ISO-size and a blocksize of 1 as arguments to dd, so it will check every single byte of the CD/thumbdrive:
`ls -l ~/Downloads/debian-live-8.2.0-amd64-standard.iso`

In the output below, we can see, that the image is 436207616 byte in size:
`-rw-r--r-- 1 jayvii users 436207616 Sep  8 22:28 debian-live-8.2.0-amd64-standard.iso`


Assuming, we burned the image to a CD, we can use *"/dev/cdrom"* as input-device. If you went with a thumbdrive, you may want to issue *"lsblk"* first, so you can determine the correct device.
`dd if=/dev/cdrom bs=1 count=436207616 | md5sum`

The output contains a hash, that can now be compared to those in the MD5SUMS-textfile.


## MD5 on Other OS's
Obviously, you can also run MD5-checksum tests on other operating systems like Windows, OSX, Solaris or BSD. However, not using any of those productively, I may only mention solutions for those very briefly.

### MD5 on Windows
There are a whole lot of possibilities for Windows. I, being a commandline fan, will introduce you to [Microsoft File Checksum Integrity Verifier](https://www.microsoft.com/en-us/download/details.aspx?id=11533), Microsoft's own solution.
This very minimalistic tool doesn't come with as many possibilities as Linux' "md5sum", however it can also check SHA-1 hashes.
After downloading and installing the tool, open up your Windows-Commandline. For this one, I will assume the ISO is located directly onto your C-Drive:
`fciv.exe C:\debian-live-8.2.0-amd64-standard.iso -md5`

### MD5 on OSX
OSX also comes witha variety of possibilities. For the sake of this tutorial, I will choose the "openssl" variante.
The terminology is very similar to Linux' md5sum, except you replace "md5sum" with "openssl md5":
`openssl md5 debian-live-8.2.0-amd64-standard.iso`

### MD5 on BSD
"GNU md5sum" and "BSD md5" are rather similar, usage wise. So replacing "md5sum" with "md5" will get you going (you can use the -q flag to suppress unimportant output):
`md5 debian-live-8.2.0-amd64-standard.iso`

### MD5 on Solaris
Solaris' "digest" also works similar to Linux' "md5sum" (you can choose the algorithm with the -a flag):
`digest -a md5 debian-live-8.2.0-amd64-standard.iso`



## Sources and referal links
* [Ubuntu: HowToMD5SUM](https://help.ubuntu.com/community/HowToMD5SUM)
* [Debian: MD5](https://wiki.debian.org/MD5)
* [Debian: Verify Authencity](https://www.debian.org/CD/verify)
* [Microsoft: File Checksum Integrity Verifier utility](https://support.microsoft.com/en-us/kb/841290)
* [GNU: DD-invocation](https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html)
* [GNU: MD5SUM-invocation](https://www.gnu.org/software/coreutils/manual/html_node/md5sum-invocation.html)
* [NIXdoc: MD5](http://nixdoc.net/man-pages/FreeBSD/man1/md5.1.html)
* [Wikipedia: MD5](https://en.wikipedia.org/wiki/MD5)
